A Mixture Model for Learning Sparse Representations

In a latent variable model, an overcomplete representation is one in which the number of latent variables is at least as large as the dimension of the data observations. Overcomplete representations have been advocated due to robustness in the presence of noise, the ability to be sparse, and an inherent flexibility in modeling the structure of data [9]. In this report, we modify factor analysis to obtain a method for learning overcomplete sparse representations by replacing the Gaussian prior on the factors with a prior that encourages sparseness. This is achieved by using the factorable Laplacian, which implicitly adds a lasso-type penalty term on the latent variables. In order to approximate the intractable integrals introduced into this model, a variational technique is used to lower bound the posterior distributions. Using this lower bound, it is possible to develop an Expectation-Maximization (EM) learning algorithm for estimating the model parameters. We use this technique to extend the sparse factor analysis model to a mixture of sparse factor analyzers and develop an EM algorithm. The new EM algorithm for the mixture model is applied to a handwritten digit recognition problem and is compared to existing methods.